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The  development  of  progress  plans  for  each  identified  tech¬ 
nical  performance  parameter  (TPP)  is  a  critical  element  of 
technical  performance  measurement.  The  measured  values 
of  TPPs  are  referred  to  as  technical  performance  measures 
(TPMs).  These  terms  are  used  interchangeably;  however,  TPMs 
more  directly  reflect  how  technical  progress  and  technical 
risk  are  measured  and  evaluated.  Progress  plans,  or  planned 
performance  profiles,  are  crucial  to  effective  risk  assessment; 
however,  methods  for  developing  these  plans  are  subjective 
in  nature,  have  no  statistical  basis  or  criteria  as  a  rule,  and 
are  not  sufficiently  addressed  in  literature.  The  methodology 
proposed  herein  for  progress  plan  development  will  involve 
the  elicitation  of  expert  judgments  to  formulate  probability 
distributions  that  reflect  the  expected  values/estimates  used 
to  establish  progress  plans. 
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The  development  of  individual  progress  plans  for  each  identified 
technical  performance  measure  (TPM)  is  a  critical  element  of  techni¬ 
cal  performance  measurement;  however,  the  methods  for  developing 
these  plans  are  subjective  in  nature,  have  no  statistical  basis  or  crite¬ 
ria  as  a  rule,  and  are  not  sufficiently  addressed  in  literature.  This  step 
is  arguably  considered  the  most  critical  aspect  of  technical  perfor¬ 
mance  measurement  because  it  provides  the  basis  for  forecasting 
successful  product  development  or  failure;  however,  the  absence  of 
clearly  defined  processes  to  develop  planned  performance  profiles 
is  a  void  that  exists  and  will  be  specifically  addressed  in  this  article. 
Bayesian  Methods  and  the  elicitation  of  expert  judgments  to  formu¬ 
late  probability  distributions  that  reflect  the  estimated  values  will  be 
utilized  to  establish  the  performance  profiles. 

The  Practical  Software  and  Systems  Measurement  (PSM)  pro¬ 
cess  presented  in  Roedler  and  Jones  (2005)  is  highly  flexible  and 
provides  the  foundation  for  the  execution  of  Technical  Performance 
Measurement.  This  process  has  been  adopted  by  the  International 
Council  of  Systems  Engineering  as  an  accepted  practice,  and 
tailoring  this  approach  for  a  typical  DoD  program  results  in  a  five- 
phase  process:  identification,  planning,  measurement,  review,  and 
reporting.  Each  phase  consists  of  multiple  process  steps,  and  the 
methodology  discussed  herein  occurs  during  planning.  Thus,  the 
latter  three  phases  are  not  addressed  or  discussed  in  this  study. 

Within  the  identification  phase,  the  customer  establishes  tech¬ 
nical  goals  and  requirements,  program  priorities  are  defined,  and 
TPMs  are  identified  to  support  program  goals  and  priorities  ideally 
traceable  to  the  lowest  level  work  breakdown  structure  (WBS)  work 
package  elements. 

The  planning  phase  begins  upon  completion  of  the  identifica¬ 
tion  phase.  During  the  planning  phase,  program  goals  are  allocated 
by  WBS  work  package  with  respect  to  budget,  schedule,  and  the 
expected  maturity  of  the  technology  under  review.  Additionally,  dur¬ 
ing  the  planning  phase,  planned  performance  profiles  and  tolerance 
bands  are  developed  for  each  TPM  identified,  initial  risk  assessments 
are  conducted,  and  a  technical  performance  baseline  is  developed. 

Why  a  Statistical  Approach  is  Needed 

Establishing  the  technical  baseline  and  individual  progress  plans 
for  each  identified  TPM  is  a  critical  element  of  technical  perfor¬ 
mance  measurement  and  is  the  means  by  which  technical  progress 
and  technical  risk  are  measured  and  evaluated.  This  occurs  during 
the  planning  phase  as  well  as  risk  assessment,  and  these  activities 
are  consistent  with  proposed  TPM  implementation  methodolo¬ 
gies  found  in  current  literature  such  as  Roedler  and  Jones  (2005). 
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Utilizing  Bayesian  Methods  and  Expert  Judgment 


However,  it  is  important  to  note  that  a  review  of  current  literature 
suggests  that  the  processes  for  the  implementation  and  execution 
of  technical  performance  measurement  on  acquisition  programs 
are  not  well  defined.  Coleman,  Kulick,  and  Pisano  (1996,  p.  6)  refer 
to  them  as  being  ad  hoc  to  a  significant  degree  due  to  the  lack  of 
formally  established  practices  and  processes  in  industry  as  well  as 
in  the  DoD.  The  PSM  guidebook  entitled  Practical  Software  and 
Systems  Measurement:  /A  Foundation  for  Objective  Project  Man¬ 
agement  (Bailey  et  a I.,  2000,  pt.  4,  p.  4-18)  refers  to  the  process 
for  developing  baseline  plans  as  “Estimation”  and  states  that  poor 
estimates  often  lead  to  failed  projects  and  result  from  the  lack  of 
systematic  estimation  processes  as  well  as  other  contributing  fac¬ 
tors.  Plans  are  developed  largely  through  consensus  by  program 
management,  developers,  systems  engineers,  and  subject  matter 
experts  who  rely  on  knowledge  from  previous  experience,  historical 
data  if  available,  and  known  cost  and/or  schedule  baselines.  These 
progress  plans  are  crucial  to  effective  risk  assessment;  however,  the 
methods  for  developing  them  are  subjective  in  nature,  have  no  sta¬ 
tistical  basis  or  criteria  as  a  rule,  and  are  not  sufficiently  addressed 
in  literature.  This  lack  of  formal  processes  to  establish  performance 
profiles  predisposes  them  to  an  inherent  degree  of  error  and  uncer¬ 
tainty,  above  and  beyond  that  inherent,  due  to  immature  technical 
development,  which  translates  into  higher  risk.  Thus,  employing  a 
formal  methodology  utilizing  an  established  statistical  approach  will 
minimize  the  level  of  uncertainty,  thereby  reducing  risk  exposure. 


Bayesian  Framework 


The  methodology  proposed  in  this  article  utilizes  an  expert 
judgment  model  within  a  Bayesian  framework  for  the  more  complex 
case  of  continuous  probability  distributions.  The  most  general  form 
of  Bayes’  Theorem  applies  to  discrete  probability  distributions,  and 
relates  the  conditional  and  prior  probabilities  of  two  events  using 
the  following  equation. 


(1) 


P(A\B)  is  considered  to  be  the  conditional  probability  of  A  given 
B;  P(B\A)  is  considered  to  be  the  likelihood  of  B  given  A,  P(A)  is  the 
prior  probability  of  A  (i.e.,  no  information  regarding  B  is  considered), 
and  P(B)  is  the  prior  or  marginal  probability  of  B. 

The  expert  judgment  model  utilized  in  this  study  for  the  case 
of  continuous  distributions  was  originally  presented  by  Mosleh  and 
Apostolakis  (1986)  to  estimate  seismic  fragility  curves  (i.e.,  the  con- 
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ditional  probability  that  a  seismic  stress  such  as  wind  or  earthquake 
will  cause  equipment  to  fail)— an  application  that,  upon  evaluation, 
proved  to  have  similarities  regarding  the  use  of  expert  opinion  elici¬ 
tation  for  planned  performance  profile  development.  In  both  cases, 
the  general  Bayesian  framework  for  continuous  distributions  uses 
the  opinions  of  experts  as  “evidence,”  and  this  evidence  is  used  as 
input  to  the  decision  maker’s  state  of  knowledge  using  Bayes’  Theo¬ 
rem.  Bayes’  Theorem,  in  its  general  form  for  continuous  probability 
distributions,  follows: 

Pr(x  I  E)  =  k  ‘L(E  I  xjPrJx)  (2) 

Prjx)  is  the  prior  state  of  knowledge  (prior  distribution)  regard¬ 
ing  the  unknown  quantity  x  prior  to  obtaining  opinions  from  the 
experts;  E  is  the  set  of  opinions  provided  by  experts  about  the  value 
of  x;  L(E  |  x)  =  the  likelihood  the  evidence  E  is  true,  given  the  true 
value  of  the  unknown  quantity  is  x;  Pr(x  \  E)  is  the  decision  maker’s 
posterior  state  of  knowledge  (posterior  distribution)  about  the 
unknown  quantity  x  given  the  set  of  opinions  E  provided  by  experts 
and  a  normalization  factor  k',  which  is  used  to  make  Pr(x  \  E)  a  prob¬ 
ability  distribution.  Within  this  framework,  the  likelihood  function 
can  be  equated  to  the  accuracy  level  of  the  expert's  estimate. 
Bayes’  Theorem  can  also  be  written  as  Equation  3  below  where  (a) 
consists  of  the  set  of  ( m )  parameters  of  the  cumulative  (unknown) 
distribution  <t>(x\a). 

Pr  (a\E)  =  k~’L(  E  \  a  )Pro  (a)  (3) 

One  unique  curve  <f>(  x  |  a )  is  specified  by  each  vector  (a),  Thus, 
the  average  of  the  infinite  number  of  distributions  0(  x  |  a )  is  defined 
by  Equation  4  below. 

0  (x)  =  la0(x  \  a  )Pr(  a\E )  da  (4) 

Assumptions 

The  following  assumptions  apply  to  the  application  of  the 
Mosleh  and  Apostolakis  (1986)  model  to  the  estimation  of  seismic 
fragility  curves  and  also  apply  to  estimation  of  progress  plan  values 
depicted  in  this  illustration. 


No.  1. 

The  unknown  distribution  being  estimated  belongs  to  a  para¬ 
metric  family  of  distributions.  This  assumption  simplifies  the 
construction  of  the  likelihood  functions.  As  a  result,  the  challenge 


A  Statistical  Approach  to  the  Development  of  Progress  Plans 
Utilizing  Bayesian  Methods  and  Expert  Judgment 


January  2011 


of  estimating  the  unknown  distribution  is  reduced  to  the  estimation 
of  its  parameters. 


No.  2. 

The  unknown  (posterior)  distribution  is  assumed  to  be  lognormal 
with  parameters  (0)  and  (co).  These  two  parameters  are  allowed  to 
vary  and  describe  the  variability  of  the  distribution.  By  definition, 
the  minimum  possible  value  for  a  lognormal  distribution  is  zero,  the 
maximum  possible  value  is  +  oo,  and  the  parameter  values  (mean  and 
standard  deviation)  must  be  greater  than  zero.  These  criteria  also 
apply  to  the  TPM  estimates  used  to  establish  a  planned  performance 
profile.  Therefore,  the  lognormal  distribution  was  deemed  the  most 
applicable  distribution  to  use  given  the  information  available. 


No.  3. 

Experts  are  independent  and  will  be  providing  independently 
assessed  percentiles.  This  assumption  is  intended  to  simplify  the 
complexity  of  the  model’s  application  to  this  problem.  A  model  for 
the  case  of  dependent  experts  is  presented  by  Mosleh  and  Apos- 
tolakis  (1986);  however,  it  will  not  be  explored  here. 


No.  4. 

Standard  deviations  assigned  to  each  expert  reflect  the  deci¬ 
sion  maker’s  level  of  confidence  in  the  expert’s  ability  to  accurately 
estimate  each  percentile.  If  available,  historical  data  reflecting  the 
planned  versus  actual  values  of  prior  predictions  should  be  used 
to  determine  the  “bias”  or  “percent  error”  to  associate  with  each 
expert’s  accuracy  and  standard  deviation. 


No.  5. 

The  percentiles  being  estimated  are  assumed  to  be  symmetric. 
Symmetric  percentiles  (10  percent,  50  percent,  and  90  percent)  have 
been  assumed  for  the  model  presented  by  Mosleh  and  Apostolakis 
(1986).  Values  for  these  percentiles  are  estimated  by  each  expert. 


No.  6. 

The  standard  deviations  of  the  percentiles  estimated  by  each 
expert  are  independent  of  the  percentiles  themselves.  In  other 
words,  each  expert  is  assumed  to  have  the  same  level  of  accu¬ 
racy  estimating  each  percentile  regardless  of  the  percentile  itself. 
Therefore,  the  standard  deviations  assigned  to  each  expert  for  each 
percentile  estimated  would  be  the  same.  However,  there  is  evidence 
to  support  that  experts  are  more  likely  to  be  less  accurate  at  higher 
percentiles,  thus  reflecting  a  greater  level  of  uncertainty  at  these 
levels  (George  &  Mensing,  1981). 
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Statistical  Approach 

The  methodology  proposed  is  presented  to  convey  the  feasi¬ 
bility  of  using  an  expert  judgment  model  developed  to  estimate 
seismic  fragility  curves  to  develop  retrospective  progress  plans. 
The  development  of  progress  plans  is  considered  to  be  a  key  and 
critical  process  element  of  technical  performance  measurement. 

The  Cockpit-21  TPM  project  described  in  the  unpublished  white 
paper  by  Coleman  et  al.  (1996)  presents  a  suitable  example  and 
adequate  data  and  information  to  convey  how  the  Mosleh  and 
Apostolakis  (1986)  methodology  could  have  been  employed  to 
develop  the  planned  performance  profiles  for  each  TPM  identified. 
The  example  utilizes  the  model  to  formulate  individual  estimates  for 
unknown  distributions  (as  opposed  to  single  values)  based  on  esti¬ 
mates  for  multiple  percentiles  provided  by  experts.  The  estimated 
distributions  reflect  values  of  specific  TPMs  for  different  points  in 
time  that  have  been  defined  to  coincide  with  key  milestone  dates. 
The  following  process  steps  are  proposed  for  the  development  of 
progress  plans  and  are  based  on  the  seismic  fragility  curve  model. 

Step  1.  Identify  TPMs. 

Step  2.  Define  dates  and  milestones  for  the  progress  plans  of  each 
parameter. 

Step  3.  Identify  experts  that  will  participate  in  the  estimation 
process. 

Step  4.  Assign  weights  to  the  experts  identified. 

Step  5.  Assign  standard  deviations  to  the  experts  identified  for  each 
percentile  estimated.  Determine  standard  deviations  for  experts 
with  unique  weights. 

Step  6.  Experts  estimate  a  TPM  value  for  each  established  percentile 
(10  percent,  50  percent,  and  90  percent)  for  each  milestone  date. 
Step  7.  Evaluate  the  standard  deviation  (a)  of  the  lognormal  param¬ 
eters  (0)  and  (ro). 

Step  8.  Evaluate  the  value  of  the  lognormal  parameters  (0)  and  (co). 
Step  9.  Evaluate  the  distribution  curve  for  each  estimate  using 
Bayes'  Theorem. 

Step  10.  Define  tolerance  bands  for  established  performance 
profiles. 

Step  11.  Plot  the  mean  values  for  each  curve  against  its  respective 
milestone  date  to  establish  the  progress  plan. 
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TABLE  1.  COCKPIT-21  TECHNICAL  PARAMETERS  AND  WEIGHTS 


Display  Electronics  Unit  (DEU)  Parameters  (50%) 

Level  1  Parameter 

wt. 

Level  2  Parameter 

Wt. 

1.  Head-Up  Display 

20% 

1.1 

HUD  Design  and 

30% 

(HUD)  Module  Status 

1.2 

Code  Status 

HUD  Full 

Qualification  Test 

(FQT)  Status 

70% 

2.  Manpower 

20% 

3.  Multifunction  Display 

20% 

3.1 

MFD  Design  and 

30% 

(MFD)  Module  Status 

Code  Status 

3.2 

MFD  FQT  Status 

70% 

4.  Requirements 

20% 

Volatility 

5.  Software  Problem 

20% 

5.1 

Software  Problem 

50% 

Reports 

5.2 

Reports  Closed 

Software  Problem 

Reports  Open 

50% 

Flight  Test  Problem  Reporting  Parameters  (50%) 

Level  1  Parameter 

wt. 

6.  Reports  Closed 

50% 

7.  Reports  Open 

50% 

Identify  Technical  Performance  Measures 


Once  goals  and  requirements  have  been  set,  TPMs  are  identified 
to  support  the  program  goals  and  priorities.  The  TPMs  identified 
are  assumed  to  have  some  notable  impact  to  program  costs,  critical 
path,  or  technical  risk.  Two  technical  areas  representing  50  percent 
each  of  the  technical  performance  baseline  were  identified  for  the 
Cockpit-21  project.  These  areas  were  the  Display  Electronics  Unit 
(DEU)  software  and  Flight  Test  Problem  Reporting.  Table  1  displays 
the  technical  parameters  and  sub-parameters  identified  for  DEU  and 
for  Flight  Test  Problem  Reporting.  Additionally,  the  weight  of  each 
parameter  at  each  level  is  shown. 

Define  Dates  and  Milestones 
for  the  Progress  Plans 

The  TPM  milestone  assessment  dates  for  the  Cockpit-21  program 
ranged  from  June  1992  through  January  of  1995,  and  the  date  range 
for  each  parameter  varied  with  respect  to  its  scheduled  develop- 
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TABLE  2.  COCKPIT-21  PROGRESS  PLAN  EVALUATION  DATES 


Display  Electronics  Unit 

Start 

Finish 

#of 

(DEU)  Parameters 

Date 

Date 

Estimates 

1.1  HUD  Design  and  Code  Status 

June  '93 

Feb  '94 

9 

1.2  HUD  Full  Qualification  Test  (FQT)  Status 

June  '93 

Feb  '94 

9 

2.  Manpower 

June  '92 

Feb  '94 

21 

3.1  MFD  Design  and  Code  Status 

June  '93 

Feb  '94 

9 

3.2  MFD  Full  Qualification  Test  (FQT)  Status 

June  '93 

Feb  '94 

9 

4.  Requirements  Volatility 

Nov  '92 

Feb  '94 

16 

5.1  Software  Problem  Reports  Closed 

Nov  ’92 

Feb  '94 

16 

5.2  Software  Problem  Reports  Open 

Oct  ’92 

Feb  '94 

17 

Flight  Test  Problem 

Start 

Finish 

#of 

Reporting  Parameters 

Date 

Date 

Estimates 

6.  Reports  Closed 

Mar  ’94 

Jan  '95 

ii 

7.  Reports  Open 

Mar  ’94 

Jan  '95 

ii 

ment  within  the  project’s  life  cycle.  Assessments  were  scheduled  at 
monthly  intervals  for  each  parameter;  however,  Roedler  and  Jones 
(2005)  indicate  that  it  is  advisable  to  utilize  significant  milestones 
and/or  design  events  to  establish  performance  profile  measure¬ 
ment  dates.  Coleman  et  al.  (1996)  also  states  that  dates  should  be 
based  on  events  during  the  development  cycle  as  opposed  to  a 
periodic  scheme.  Therefore,  this  would  be  the  preferred  method  for 
establishing  evaluation  dates  for  TPM  values  when  implementing 
technical  performance  measurement  on  future  projects.  The  range 
of  dates  and  number  of  estimates  associated  with  each  progress 
plan  developed  for  the  Cockpit-21  program  are  depicted  in  Table  2. 
Since  TPMs  are  tracked  at  the  lowest  level,  it  is  only  necessary  to 
develop  performance  profiles  for  the  lowest  level  parameter.  Thus, 
for  the  purpose  of  this  application,  progress  plans  are  not  developed 
for  level  2  parameters  that  have  level  3  subparameters;  those  level 
2  parameters  are  not  depicted  in  Table  2. 

Identify  Experts 

The  experts  involved  in  the  TPM  planning  process  on  a  DoD 
program  will  most  often  be  the  members  of  the  integrated  product 
teams  (IPTs)  established  to  test,  monitor,  and  evaluate  technical 
progress.  IPTs  will  primarily  consist  of  contractors/developers. 
However,  government  representatives  may  work  with  contractors  to 
share  responsibilities  pertaining  to  the  planning  process.  Members 
of  each  IPT  are  selected  based  on  their  knowledge  and  experience 
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regarding  previous  development  efforts  with  the  same  or  similar 
technology  and  may  be  systems  engineers,  test  engineers,  research¬ 
ers,  or  analysts.  For  the  purpose  of  this  study,  the  IPT  members  are 
considered  to  be  the  experts,  and  the  IPT  leads  are  considered  to 
be  more  knowledgeable  than  the  other  members  of  the  team. 

The  question  of  how  many  experts  are  optimal  was  addressed 
by  K.  Walker  (personal  communication,  November  11,  2004).  She 
concluded  there  has  been  no  evidence  of  an  existing  argument  for 
an  ideally  statistically  based  sample  size.  Her  literature  review  of  38 
studies  revealed  90  percent  used  11  or  less  experts,  and  referenced 
Steve  Hora,  who  has  been  repeatedly  quoted  for  his  argument  of 
“three  and  seldom  more  than  six”  is  sufficient  (Hora,  2004,  p.  5;  K. 
Walker,  personal  communication,  November  11,  2004).  The  example 
presented  in  this  study  assumes  four  experts  are  participating  in  the 
progress  plan  development  process  as  members  of  an  IPT. 

Assign  Weights  to  the  Experts 

Weights  are  assigned  by  the  decision  maker  to  each  expert 
providing  an  estimate  as  a  means  to  place  more  or  less  value  on 
the  responses  of  experts  that  are  assumed  to  have  greater  or  lesser 
accuracy  with  respect  to  their  estimation  ability.  The  more  confi¬ 
dence  a  decision  maker  has  in  a  particular  expert’s  ability,  the  more 
weight  the  expert’s  estimate  should  carry  with  respect  to  other 
experts.  The  weights  of  all  experts  should  be  normalized  so  that 
they  sum  to  one.  If  available,  this  step  is  where  it  would  be  appro¬ 
priate  to  review  historical  data  from  previous  projects  to  evaluate 
the  estimation  accuracy  of  experts  conducting  the  estimates  to 
determine  the  appropriate  weighting  scheme.  Additionally,  the 
experience  level  of  each  expert  (or  IPT  member)  and  their  level  of 
knowledge  regarding  the  technical  area  being  evaluated  should  be 
considered  when  assigning  weights.  The  assessment  by  the  deci¬ 
sion  maker  in  this  example  assumes  each  expert  is  equally  weighted 
with  the  exception  of  the  IPT  lead.  The  lead  is  given  more  weight  to 
compensate  for  additional  knowledge  and  experience  in  the  techni¬ 
cal  area  of  interest. 

Assign  Standard  Deviations  to  the  Experts 

Standard  deviations  are  assigned  by  the  decision  maker  to 
each  percentile  estimated  by  each  expert.  Each  standard  deviation 
reflects  the  perceived  range  of  deviation  from  the  true  value  being 
estimated  at  that  percentile.  The  approach  to  assigning  standard 
deviations  is  similar  to  that  of  assigning  weights  in  that  historical 
data  from  previous  projects  should  be  reviewed,  if  available,  to 
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evaluate  the  true  estimation  accuracy  or  percentage  error  realized 
by  experts  with  prior  predictions.  This  should  ensure  the  proper 
calibration  of  the  experts  and  allow  the  decision  maker  to  account 
for  overconfidence  of  the  experts  as  discussed  by  Hora  (2004),  and 
errors  of  estimation  as  discussed  by  Winkler  (1981).  In  the  case  of 
the  Cockpit-21  project,  data  from  previous  TPM  pilot  implementation 
projects  discussed  by  Coleman  et  al.  (1996),  such  as  the  Air  Deploy¬ 
able  Active  Receiver  (ADAR)  sonobuoy  development  program  and 
the  LAMPS  Block- 1 1  Upgrade  program,  would  serve  as  appropriate 
reference  materials  for  this  effort.  The  model  presented  in  this  study 
assumes  the  standard  deviations  of  all  experts  are  equal  with  the 
exception  of  the  IPT  lead,  who  has  a  greater  weight  than  the  remain¬ 
ing  team  members  (i.e.,  oIK  =  oK)  where  (J)  equals  the  Jth  percentile 
and  (K)  equals  th eKth  expert.  To  calculate  the  standard  deviation  of 
the  IPT  lead,  Equation  5a  must  be  used  where  N  =  the  total  number 
of  experts,  oK  =  the  standard  deviation  of  expert  (K),  and  w  =  the 
weight  of  expert  (K). 

N  a;2 

Zctk2  = -  (5a) 

K=1  W] 

This  will  result  in  the  standard  deviation  of  the  IPT  lead  being 
less  than  the  remaining  IPT  members  due  to  the  lead  being  assigned 
a  heavier  weight. 

Estimate  TPM  Values  by  Percentile 

The  methodology  proposed  here  requires  TPM  values  to  be 
estimated  by  each  expert  for  each  established  percentile  (10  per¬ 
cent,  50  percent,  and  90  percent)  by  milestone  date.  Meaning,  each 
expert  must  predict  the  true  value  that  has  a  10  percent  likelihood 
of  being  observed,  a  50  percent  likelihood  of  being  observed,  and 
a  90  percent  likelihood  of  being  observed  for  the  given  evaluation 
date.  Experts  formulate  estimates  based  on  the  information  they 
have  available  to  them  at  the  time,  and  this  information  may  include 
historical  and/or  current  test  data,  the  results  of  formal  functional 
analysis,  and  expert  opinion  based  on  knowledge  and  experience. 

Table  3  depicts  the  proposed  estimations  for  the  first  milestone 
date  of  the  first  technical  parameter  shown  in  Table  2  (Display  1.1, 
HUD  Design  and  Code  Status),  the  basis  for  the  response  provided, 
as  well  as  the  assigned  weight  and  standard  deviation  for  each 
expert.  Equation  5a  was  used  to  estimate  (o4)  as  follows: 
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TABLE  3.  HUD  DESIGN  AND  CODE  STATUS  EXPERT  OPINION  DATA 


Expert 

HUD  Design  and  Code  Status  -  June  1993 

Expert  Expert  std.  Percentiles  Basis  for 

wt.  dev.  (ok)  10%  50%  90%  Response 

i 

.24 

0.100 

22.0 

23.0  26.0 

test  data,  analytical 

methods 

2 

.24 

0.100 

21.5 

22.8  24.0 

test  data,  analytical 
methods,  expert 
opinion 

3 

.24 

0.100 

22.0 

23.5  24.5 

test  data,  analytical 

methods 

4 

.28 

0.093 

21.5 

24.0  25.0 

test  data,  analytical 
methods,  expert 
opinion 

Using  the  values  in  Table  2,  it  is  evident  that  each  expert  must 
provide  27  unique  estimates  (9  estimates  x  3  percentiles)  to  estab¬ 
lish  a  progress  plan  for  this  parameter;  and  a  total  of  12  estimates  (3 
percentiles  x  4  experts)  will  be  used  in  the  expert  judgment  model 
to  establish  the  unknown  distribution  for  each  individual  estimate. 
To  complete  the  expert  judgment  analysis  for  all  parameters  and  all 
evaluation  dates,  this  process  would  be  completed  for  each  param¬ 
eter  shown  in  Table  2,  resulting  in  each  expert  providing  378  unique 
estimates  (126  estimate  dates  x  3  percentiles  per  date). 

Evaluate  the  Standard  Deviation 
of  the  TPM  Values 

The  unknown  distribution  associated  with  each  estimate  is 
assumed  to  be  lognormal  with  parameters  (a)  and  (to).  The  Mosleh 
and  Apostolakis  (1986)  model  uses  Equations  6  and  7  to  evaluate 
the  values  of  the  standard  deviations  for  (0)  and  (co).  The  standard 
deviation  for  (0)  is  denoted  by  Equation  6. 
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(6) 


K=1 


o  reflects  the  standard  deviation  for  (©).  The  standard  devia¬ 
tion  for  (co)  is  denoted  by  Equation  7,  where  am  reflects  the  standard 
deviation  of  (co)  and  Zgo  =  1.285  from  tables  of  the  standard  normal 
distribution. 


-1 


N 


(7) 


K=l 


Evaluate  the  Value  of  the  Lognormal  Parameters 

The  unknown  distribution  associated  with  each  estimate  is 
assumed  to  be  lognormal  with  parameters  (0)  and  (co).  The  Mosleh 
and  Apostolakis  (1986)  model  uses  Equations  8  and  9  to  evaluate 
the  values  of  the  parameters  (0)  and  (co).  The  parameter  value  for  (0) 
is  denoted  by  Equation  8. 


N 


(8) 


K=1 


The  parameter  value  for  (co)  is  denoted  by  Equation  9. 


N 


°>m  =  1  ^WK  (InX90K~  lnXWK> 


(9) 


Evaluate  Each  Unknown  Distribution 
Using  Bayes'  Theorem 


Now  that  weights  and  standard  deviations  have  been  assigned 
to  the  experts,  values  have  been  estimated  for  each  percentile  by 
the  experts,  the  standard  deviations  of  the  parameter  values  have 
been  evaluated,  and  the  parameter  values  themselves  have  been 
evaluated,  all  requisite  information  is  available  to  evaluate  the  value 
of  the  unknown  (posterior)  distribution  for  the  parameter  depicted 
in  Table  3.  Using  the  Mosleh  and  Apostolakis  model  for  Bayes’  Theo¬ 
rem,  the  posterior  distribution  can  be  evaluated  using  Equation  10. 
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Upon  evaluation  of  Equation  10  for  the  parameter  depicted  in 
Table  3,  and  assuming  the  same  experts  are  providing  all  estimates, 
steps  7  through  9  are  repeated  for  the  remaining  parameters  identi¬ 
fied  in  Table  2  to  complete  the  expert  judgment  analysis. 

Define  Tolerance  Bands  for  Performance  Profiles 

Following  completion  of  the  expert  judgment  analysis,  the  next 
step  is  to  define  tolerance  bands.  Tolerance  bands  reflect  the  allow¬ 
able  range  of  variation  and  level  of  acceptable  risk  for  a  defined 
TPM  estimate  on  a  given  milestone  date.  They  alert  management 
that  actions  may  be  necessary  to  get  the  TPM  back  on  track.  Actual 
values  that  exceed  the  allowable  range  (e.g.,  20  percent)  for  any 
TPM  estimate  denote  high  risk  (red)  and  will  trigger  management 
intervention.  Additionally,  allowable  ranges  are  defined  for  low-risk 
items  (green),  and  medium-risk  items  (yellow)  as  well.  Typically,  the 
bands  depicted  on  a  performance  profile  represent  the  maximum 
allowable  variation. 

Plot  the  Expected  Values  for  Each  Curve 

If  the  expected  values  defined  in  the  Cockpit-21  white  paper 
are  assumed  to  be  the  mean  expected  values  determined  using 
the  expert  judgment  methodology,  these  values,  along  with  the 
tolerance  bands  and  the  threshold  values,  would  establish  the 
performance  baseline  for  each  performance  measure  when  plot¬ 
ted  against  their  respective  milestone  dates.  The  Figure  depicts 
the  expected  values  used  to  define  the  individual  progress  plan 
for  HUD  Design  and  Code  and  the  corresponding  dates  for  each 
estimate.  Actual  values,  a  revised  plan,  and  the  threshold  value  are 
also  plotted  in  the  example  shown  for  comparison  with  the  baseline 
expected  values. 


Discussion 

This  article  describes  how  expert  judgment  can  be  utilized 
within  a  Bayesian  framework  to  develop  a  formal  statistical  model 
to  quantify  expert  opinions  as  probability  distributions  for  the 
purpose  of  establishing  TPM  progress  plans.  To  demonstrate  these 
attributes,  key  assumptions  were  made,  and  actual  TPM  data  were 
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FIGURE.  HUD  DESIGN  AND  CODE  PROGRESS  PLAN  EXAMPLE 
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used  to  illustrate  the  application  of  the  methodology.  Baseline 
progress  plans  are  crucial  to  effective  risk  assessment;  however, 
the  methods  for  developing  these  plans  for  each  parameter  are 
subjective  in  nature,  have  no  statistical  basis  or  criteria  as  a  rule, 
and  are  not  sufficiently  addressed  in  literature.  This  lack  of  formal 
processes  to  establish  performance  profiles  predisposes  them  to 
an  inherent  degree  of  error  and  uncertainty  (i.e.,  risk).  The  formal 
methodology  presented  in  this  article  offers  an  alternative  that  will 
arguably  produce  more  accurate  progress  plans  and  will  minimize 
the  level  of  uncertainty,  thereby  resulting  in  reduced  risk  exposure. 
This  proposed  methodology  provides  program  management  with 
another  decision-making  tool  that  can  be  used  to  strengthen  estab¬ 
lished  systems  engineering  processes. 
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